Open Problem: The Oracle Complexity of Smooth Convex Optimization in Nonstandard Settings

First-order convex minimization algorithms are currently the methods of choice for large-scale sparse – and more generally parsimonious – regression models. We pose the question on the limits of performance of black-box oriented methods for convex minimization in non-standard settings, where the regularity of the objective is measured in a norm not necessarily induced by the feasible domain. This question is studied for `p/`q-settings, and their matrix analogues (Schatten norms), where we find surprising gaps on lower bounds compared to state of the art methods. We propose a conjecture on the optimal convergence rates for these settings, for which a positive answer would lead to significant improvements on minimization algorithms for parsimonious regression models. Let (E, ‖ · ‖) be a finite-dimensional normed space. Given parameters, 1 < κ ≤ 2, and L > 0, we consider the class F‖·‖(κ, L) of convex functions that are (κ, L)-smooth w.r.t. norm ‖ · ‖. One such function f : E→ R satisfies that ‖∇f(y)−∇f(x)‖∗ ≤ L‖x− y‖κ−1 ∀x, y ∈ E. Notice that the case κ → 1 corresponds essentially to nonsmooth (Lipschitz continuous) convex functions, and κ = 2 corresponds to smooth (with Lipschitz continuous gradients) convex functions. Given a convex body X ⊆ E, we are interested on the complexity of the problem class P = (F‖·‖(κ, L), X), comprised of optimization problems with objective f ∈ F‖·‖(κ, L) Opt(f,X) = min{f(x) : x ∈ X}. (Pf,X) We study a black-box oracle model where most algorithms based on subgradient computations can be implemented1. Here, an algorithm is allowed to perform queries x ∈ E, and for any such query the oracle returns Of (x) (e.g., for first-order methods, Of (x) = ∇f(x)). The only assumption on the oracle is locality: For any x ∈ E and f, g ∈ F such that there exists a neighborhood B(x, δ) where f ≡ g, then Of (x) = Og(x). Given T > 0, we consider an algorithm A whose output xT (A, f) is only determined by T (adaptive) oracle queries. We define the accuracy of algorithm A on an instance f as ε(A, f) := f(xT (A, f)) − Opt(f,X), if xT (A, f) ∈ X , otherwise ε(A, f) = ∞. We characterize optimal 1. Notable exceptions are methods exploiting explicit saddle-point description, e.g., the smoothing technique by Nesterov (2005). Note however that such algorithms do not give improvement in the smooth case.